Space reclamation in asynchronously mirrored space-efficient secondary volumes

ABSTRACT

A method for releasing storage space in asynchronously mirrored space-efficient secondary volumes is disclosed. In one embodiment, such a method includes reading a first copy of a free-space data structure stored on a space-efficient secondary volume. The free-space data structure tracks the usage status of storage elements in the space-efficient secondary volume. The method analyzes the first copy to determine which storage elements in the space-efficient secondary volume are not being used. Upon completion of a consistency group on the space-efficient secondary volume, the method reads a second copy of the free-space data structure and compares the first copy to the second copy to determine which storage elements had their usage status change during analysis of the first copy. The method releases storage elements in the space-efficient secondary volume that are not being used. A corresponding system and computer program product are also disclosed.

BACKGROUND

Field of the Invention

This invention relates to systems and methods for reclaiming space inasynchronously mirrored space-efficient secondary volumes.

Background of the Invention

On storage systems such as the IBM DS8000™ enterprise storage system,space-efficient volumes may be used to more efficiently utilize storagespace. A space-efficient volume differs from a standard volume in thatdata is not physically stored in the volume. Rather, the space-efficientvolume is a virtual volume whose data is physically stored in a commonrepository. A mapping structure keeps track of where a space-efficientvolume's data is physically located in the repository. Stated otherwise,the mapping structure maps logical tracks of the space-efficient volumeto physical tracks of the repository. From the perspective of a hostdevice or other external system, reading from or writing to aspace-efficient volume may be the same as reading from or writing to astandard volume.

While physical storage space may be allocated to space-efficient volumeswhen needed, the same physical storage space may be reclaimed from thespace-efficient volumes when it is no longer needed. Currently, whentracks are deleted by host-system-based software, the DS8000™ storagecontroller relies on updates to its metadata to release the tracks froma space-efficient volume. More specifically, the DS8000™ storagecontroller relies on certain metadata associated with the tracks to beoverwritten with zeros or marked to indicate that the tracks need to beremoved from the space-efficient volume. This metadata is later scannedby the storage controller to identify which tracks should be removed.Once identified, the storage controller may add the tracks to a freestorage pool to be used for future allocations. Over time, events suchas errors or miscommunications may occur which may result in therequired metadata not being zeroed out correctly. For example, ifsoftware on a host system deletes a data set or particular tracks of adata set, the software on the host system may notify the storagecontroller so that the storage controller can release the tracks backinto the free storage pool. If the notifications from the software arelost or not properly registered or handled by the storage controller,the tracks may not be released back into the free storage pool. Overtime this may result in storage space in the storage system that is notutilized, but nevertheless tied up and unavailable for use.

In view of the foregoing, what are needed are systems and methods tomore effectively reclaim space in space-efficient volumes. Ideally, suchsystems and methods will prevent situations where tracks, even thoughdeleted or marked as unused by a host system, are not released to a freestorage pool by the storage controller. Yet further needed are systemsand methods to reclaim space in space-efficient volumes used insynchronous and asynchronous data replication environments. For example,systems and methods are needed to reclaim space in asynchronouslymirrored space-efficient secondary volumes.

SUMMARY

The invention has been developed in response to the present state of theart and, in particular, in response to the problems and needs in the artthat have not yet been fully solved by currently available systems andmethods. Accordingly, the invention has been developed to providesystems and methods to release storage space in space-efficientsecondary volumes. The features and advantages of the invention willbecome more fully apparent from the following description and appendedclaims, or may be learned by practice of the invention as set forthhereinafter.

Consistent with the foregoing, a method for releasing storage space inasynchronously mirrored space-efficient secondary volumes is disclosed.In one embodiment, such a method includes reading a first copy of afree-space data structure stored on a space-efficient secondary volume.The free-space data structure tracks the usage status of storageelements in the space-efficient secondary volume. The method analyzesthe first copy to determine which storage elements in thespace-efficient secondary volume are not being used. The method thenwaits for a consistency group on the space-efficient secondary volume tocomplete. Upon completion of the consistency group, the method reads asecond copy of the free-space data structure and compares the first copyto the second copy to determine which storage elements had their usagestatus change during analysis of the first copy. The method releasesstorage elements in the space-efficient secondary volume that are notbeing used.

A corresponding system and computer program product are also disclosedand claimed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered limiting of its scope, the embodiments of the inventionwill be described and explained with additional specificity and detailthrough use of the accompanying drawings, in which:

FIG. 1 is a high-level block diagram showing one example of a networkarchitecture in which systems and methods in accordance with theinvention may be implemented;

FIG. 2 is a high-level block diagram showing one example of a storagesystem hosting one or more space-efficient volumes;

FIG. 3 is a high-level block diagram showing logical volumes exposed bythe storage system, and particularly showing a volume table of contents(VTOC) and data sets stored on a logical volume;

FIG. 4 is a high-level block diagram showing use of space-efficientlogical volumes for storing point-in-time copies;

FIG. 5 is a high-level block diagram showing use of a free space bitmapby both a host system and storage controller;

FIG. 6 is a high level block diagram showing one embodiment of asynchronous data replication system comprising primary and secondarystorage systems;

FIG. 7 is a high level block diagram showing various sub-modules that beincluded in a space reclamation module used on a secondary storagesystem of the synchronous data replication system;

FIG. 8 is a process flow diagram showing one embodiment of a method forreclaiming space in secondary storage volumes in a synchronous datareplication system;

FIG. 9 is a high level block diagram showing one embodiment of anasynchronous data replication system comprising primary and secondarystorage systems;

FIG. 10 is a high level block diagram showing various sub-modules thatbe included in a space reclamation module for reclaiming space inasynchronously mirrored space-efficient secondary volumes; and

FIG. 11 is a process flow diagram showing one embodiment of a method forreclaiming space in asynchronously mirrored space-efficient secondaryvolumes.

DETAILED DESCRIPTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the Figures herein,could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the invention, as represented in the Figures, is notintended to limit the scope of the invention, as claimed, but is merelyrepresentative of certain examples of presently contemplated embodimentsin accordance with the invention. The presently described embodimentswill be best understood by reference to the drawings, wherein like partsare designated by like numerals throughout.

The present invention may be embodied as a system, method, and/orcomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The computer readable storage medium may be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on a user's computer,partly on a user's computer, as a stand-alone software package, partlyon a user's computer and partly on a remote computer, or entirely on aremote computer or server. In the latter scenario, a remote computer maybe connected to a user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider). In some embodiments,electronic circuitry including, for example, programmable logiccircuitry, field-programmable gate arrays (FPGA), or programmable logicarrays (PLA) may execute the computer readable program instructions byutilizing state information of the computer readable programinstructions to personalize the electronic circuitry, in order toperform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, may be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus, or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

Referring to FIG. 1, one example of a network architecture 100 isillustrated. The network architecture 100 is presented to show oneexample of an environment where systems and methods in accordance withthe invention may be implemented. The network architecture 100 ispresented only by way of example and not limitation. Indeed, the systemsand methods disclosed herein may be applicable to a wide variety ofnetwork architectures, in addition to the network architecture 100shown.

As shown, the network architecture 100 includes one or more computers102, 106 interconnected by a network 104. The network 104 may include,for example, a local-area-network (LAN) 104, a wide-area-network (WAN)104, the Internet 104, an intranet 104, or the like. In certainembodiments, the computers 102, 106 may include both client computers102 and server computers 106 (also referred to herein as “host systems”106). In general, the client computers 102 initiate communicationsessions, whereas the server computers 106 wait for requests from theclient computers 102. In certain embodiments, the computers 102 and/orservers 106 may connect to one or more internal or externaldirect-attached storage systems 109 (e.g., arrays of hard-disk drives,solid-state drives, tape drives, etc.). These computers 102, 106 anddirect-attached storage systems 109 may communicate using protocols suchas ATA, SATA, SCSI, SAS, Fibre Channel, or the like.

The network architecture 100 may, in certain embodiments, include astorage network 108 behind the servers 106, such as astorage-area-network (SAN) 108 or a LAN 108 (e.g., when usingnetwork-attached storage). This network 108 may connect the servers 106to one or more storage systems, such as arrays 110 of hard-disk drivesor solid-state drives, tape libraries 112, individual hard-disk drives114 or solid-state drives 114, tape drives 116, CD-ROM libraries, or thelike. To access a storage system 110, 112, 114, 116, a host system 106may communicate over physical connections from one or more ports on thehost 106 to one or more ports on the storage system 110, 112, 114, 116.A connection may be through a switch, fabric, direct connection, or thelike. In certain embodiments, the servers 106 and storage systems 110,112, 114, 116 may communicate using a networking standard such as FibreChannel (FC).

Referring to FIG. 2, one embodiment of a storage system 110 containingan array of hard-disk drives 204 and/or solid-state drives 204 isillustrated. As shown, the storage system 110 includes a storagecontroller 200, one or more switches 202, and one or more storagedevices 204, such as hard disk drives 204 or solid-state drives 204(such as flash-memory-based drives 204). The storage controller 200 mayenable one or more hosts 106 (e.g., open system and/or mainframe servers106 running operating systems such as MVS, z/OS, or the like) to accessdata in the one or more storage devices 204.

In selected embodiments, the storage controller 200 includes one or moreservers 206. The storage controller 200 may also include host adapters208 and device adapters 210 to connect the storage controller 200 tohost devices 106 and storage devices 204, respectively. Multiple servers206 a, 206 b may provide redundancy to ensure that data is alwaysavailable to connected hosts 106. Thus, when one server 206 a fails, theother server 206 b may pick up the I/O load of the failed server 206 ato ensure that I/O is able to continue between the hosts 106 and thestorage devices 204. This process may be referred to as a “failover.”

In selected embodiments, each server 206 may include one or moreprocessors 212 and memory 214. The memory 214 may include volatilememory (e.g., RAM) as well as non-volatile memory (e.g., ROM, EPROM,EEPROM, hard disks, flash memory, etc.). The volatile and non-volatilememory may, in certain embodiments, store software modules that run onthe processor(s) 212 and are used to access data in the storage devices204. The servers 206 may host at least one instance of these softwaremodules. These software modules may manage all read and write requeststo logical volumes in the storage devices 204.

One example of a storage system 110 having an architecture similar tothat illustrated in FIG. 2 is the IBM DS8000™ enterprise storage system.The DS8000™ is a high-performance, high-capacity storage controllerproviding disk storage that is designed to support continuousoperations. Nevertheless, the apparatus and methods disclosed herein arenot limited to operation with the IBM DS8000™ enterprise storage system110, but may operate with any comparable or analogous storage system110, regardless of the manufacturer, product name, or components orcomponent names associated with the system 110. Furthermore, any storagesystem that could benefit from one or more embodiments of the inventionis deemed to fall within the scope of the invention. Thus, the IBMDS8000™ is presented only by way of example and is not intended to belimiting.

Referring to FIG. 3, in certain embodiments, a storage system 110 suchas that illustrated in FIG. 2 may be configured to present or expose oneor more volumes 300 to a host system 106. The volumes 300 may be logicalvolumes 300, meaning that the volumes 300 may appear to be physicaldrives 204 (e.g., hard drives, solid state drives, etc.) to a hostsystem 106 but do not necessarily directly correlate to physical drives204 on the storage system 110. For example, in certain embodiments, aphysical drive 204 may be used by more than one logical volume 300 or alogical volume 300 may span all or part of multiple physical drives 204.A storage virtualization layer 302 within the storage system 110 or mayexpose the logical volumes 300 and handle the mapping between thelogical volumes 300 and the physical drives 204.

As further shown in FIG. 3, in certain embodiments, each logical volume300 may store a volume table of contents (VTOC) 304 and one or more datasets 306. A VTOC 304 may contain information for locating data sets 306on the associated logical volume 300. In certain embodiments, the VTOC304 is located at the beginning of the logical volume 300 and may listthe names of each data set 306 on the logical volume 300 as well as thedata set's size, location, and permissions. The VTOC 304 may also storeinformation describing each area of contiguous free space in the logicalvolume 300. The VTOC 304 is typically created at the time the logicalvolume 300 is initialized.

To access a particular data set 306 on a storage system 110, a host 106may query a host-based catalog to determine the logical volume 300 onwhich the data set 306 resides. Once the correct logical volume 300 isdetermined, the host 106 locates the VTOC 304 on the logical volume 300and searches the VTOC 304 to determine where the data set 306 is stored.The host 106 may then access the data set 306 at the determinedlocation.

In general, a host 106 and host-system-based software is able torecognize, understand, and utilize the VTOC 304 to access data sets 306on the logical volume 300. However, the storage controller 200 hostingthe logical volume 300 typically will not recognize or understand theVTOC 304. Instead, the storage controller 200 may store and maintaininternal metadata 216 (See FIG. 2) in its memory 214 to understand theconfiguration of a logical volume 300.

Referring to FIG. 4, on storage systems such as the IBM DS8000™enterprise storage system, FlashCopy is a function used to create nearlyinstantaneous point-in-time copies of volumes 300 or data sets 306 on astorage system 110. These point-in-time copies may be used for backupsor other purposes. Once created, the point-in-time copies areimmediately available for both read and write access to host systems106. Space Efficient FlashCopy is a function similar to conventionalFlashCopy except that space-efficient volumes 300 created with thisfunction are not allocated all their defined storage space at the timeof creation. Rather, storage space is allocated when data is actuallywritten to a space-efficient volume 300.

FIG. 4 shows a scenario where space-efficient logical volumes 300 (shownwith the dotted lines) are created to store point-in-time copies of datain corresponding source logical volumes (shown with solid lines). Whenthe space-efficient logical volumes 300 are created, no physical spacemay be allocated to the space-efficient logical volumes 300. Rather,space may be allocated to the space-efficient logical volumes 300 on anas-need basis. For example, when new data is written to a track 400 a ofa source logical volume 300, the old data in the track 400 a may becopied over to the associated space-efficient logical volume 300 inorder to preserve the point-in-time copy of the data.

When the data is copied over, a track 400 b may be allocated to thespace-efficient logical volume 300 from a repository 402 of free tracksin order to store the old data. In this way, tracks may be allocated tospace-efficient logical volumes 300 if and when they are needed,providing more efficient utilization of storage space. This also allowsthe space-efficient logical volumes 300 to be over-provisioned, meaningthat the space-efficient logical volumes 300 may be logically largerthan the amount of physical storage space backing them.

When a track is no longer being used in a space-efficient logical volume300, such as when a track of data or an entire data set 306 has beendeleted from the space-efficient logical volume 300, the track may bereleased to the repository 402 so that it can be used again either inthe same or different space-efficient logical volume 300. Inconventional implementations, if a host system 106 or host-system-basedsoftware deletes data, a volume space manager 508 (See FIG. 5) on thehost system 106 updates the VTOC 304 to indicate that the storage spaceused to store the data is no longer in use. The volume space manager 508then notifies the storage controller 200 that the storage space is nolonger being used. A space reclamation module 510 (See FIG. 5) on thestorage controller 200 may, in turn, update the track or tracks formerlyused to store the data (such as by writing all zeros to the track ortracks) or update the internal metadata 216 of the storage controller200 to indicate that the tracks are no longer in use. The spacereclamation module 510 may then periodically scan the tracks and/or themetadata 216 to determine which tracks have been zeroed out or marked,and then release the tracks to the repository 402 so that they can beused again.

Unfortunately, scenarios may occur where storage space that is freed bya volume space manager 508 on the host system 106 is not released to therepository 402 by the storage controller 200. Because the spacereclamation module 510 on the storage controller 200 relies onnotifications from the volume space manager 508 on the host system 106to identify tracks to release to the repository 402, these tracks maynot be released if the notifications are not received or loggedcorrectly by the storage controller 200. Lost or incorrectly loggednotifications may be the result of errors, miscommunications, codeglitches, or the like. Over time, tracks that are not properly releasedmay result in storage space in the storage system 110 that is notutilized, but nevertheless tied up and unavailable for use.

Referring to FIG. 5, in certain embodiments in accordance with theinvention, a new data structure may be created that allows a host system106 and storage controller 200 to coordinate the release of storagespace back to the repository 402. In certain embodiments, this new datastructure may be stored in the VTOC 304 previously discussed. Since astorage controller 200 may not normally recognize and/or understand aVTOC 304, various mechanisms may be put in place to enable a spacereclamation module 510 on the storage controller 200 to identify andutilize the new data structure in the VTOC 304. In other embodiments,the new data structure is located on a space efficient logical volume300 outside of a VTOC 304, such as in a data set 306 or other locationexternal to the VTOC 304.

In certain embodiments, the new data structure is a bitmap 506, whereeach bit in the bitmap 506 represents a storage area in aspace-efficient logical volume 300. For example, each bit in the bitmap506 may represent a track in the space-efficient logical volume 300.When a host system 106 or host-system-based software uses or releases atrack, the volume space manager 508 on the host system 106 may mark thecorresponding bit in the bitmap 506 to indicate that the track is eitherused or unused. The space reclamation module 510 on the storagecontroller 200 may, in turn, periodically scan the bitmap 506 todetermine which tracks to release to the repository 402. If a bit for atrack is marked as unused, the space reclamation module 510 may releasethe corresponding track to the repository 402. In certain embodiments,the bitmap 506 is embodied as a new type of data set control block(DSCB) 500 in the VTOC 304. However, unlike other data set controlblocks 500 (e.g., data set descriptor DSCBs 502, free space descriptorDSCBs 504, etc.), the new bitmap 506 may be recognized and utilized notonly by a host system 106, but also by a storage controller 200.

When a volume space manager 508 on a host system 106 changes the statusof a track in a logical volume 300, such as by changing the track statusfrom used to unused, or unused to used, the volume space manager 508 mayupdate the bitmap 506 to reflect the new status. To accomplish this, thevolume space manager 508 may acquire a lock on the track storing thebitmap 506 to ensure that the storage controller 200 or other processescan't access the bitmap 506 while updates are being made. The volumespace manager 508 may then update the bitmap 506 to reflect the newstatus and release the lock when the operation is complete. The spacereclamation module 510 in the storage controller 200 may likewiseacquire a lock on the bitmap 506 when the bitmap 506 is analyzed orupdated. This will ensure that neither the host system 106 nor storagecontroller 200 overwrites a pending operation. If the storage controller200 attempts to access the bitmap 506 while the host system 106 has alock, the storage controller 200 may need to wait until the lock isreleased to access the bitmap 506, and vice versa.

As mentioned above, various mechanisms may be put in place to enable astorage controller 200 to find and utilize the bitmap 506 on aspace-efficient logical volume 300. In one embodiment, the bitmap 506 iscreated and registered with the storage controller 200 at the time aspace-efficient logical volume 300 is initialized. This will allow thestorage controller 200 to store the location of the bitmap 506 in itsinternal metadata 216 so that unused tracks can be determined andreleased. If a logical volume 300 is reinitialized in a way that changesthe location of the bitmap 506, the bitmap 506 may be reregistered withthe storage controller 200.

In another embodiment, a unique identifier is stored on thespace-efficient logical volume 300 b with the bitmap. This may enable astorage controller 200 to locate the bitmap by scanning the logicalvolume 300 for the unique identifier. In yet another embodiment, thestorage controller 200 may be updated to have knowledge of the VTOC 304,there enabling the storage controller 200 to scan the VTOC 304 andlocate the bitmap 506. Other techniques for enabling the storagecontroller 200 to locate and identify the bitmap 506 are possible andwithin the scope of the invention.

Referring to FIG. 6, although the systems and methods described hereinhave been discussed in relation to space-efficient FlashCopy, a spacereclamation module 510 in accordance with the invention may be used toreclaim space in space-efficient volumes 300 more generally. In certainembodiments, a space reclamation module 510 in accordance with theinvention may be used in data replication systems, such as synchronousdata replication systems, or peer-to-peer remote copy (PPRC) systems, toreclaim space in space-efficient volumes 300 stored thereon. FIG. 6shows one embodiment of a synchronous data replication system 600, orPPRC system 600, comprising a primary storage system 110 a and secondarystorage system 110 b. In such a system 600, writes originating from ahost system 106 will be written to the primary storage system 110 a andmirrored to the secondary storage system 110 b. A write is onlyconsidered complete when it has completed to both the primary storagesystem 110 a and secondary storage system 110 b.

For example, when performing a write operation in such a system 600, ahost system 106 may send a write request to the primary storage system110 a. The write operation may be performed on the primary storagesystem 110 a. The primary storage system 110 a may, in turn, transmit awrite request to the secondary storage system 110 b. The secondarystorage system 110 b may execute the write operation and return a writeacknowledge signal to the primary storage system 110 a. Once the writehas completed on both the primary and secondary storage systems 110 a,110 b, the primary storage system 110 a returns a write acknowledgesignal to the host system 106. The I/O is considered complete when thehost system 106 receives the write acknowledge signal.

As shown, a space reclamation module 510 a may be present on the primarystorage system 110 a to reclaim unused space in space-efficient primaryvolumes 300 a. Similarly, a space reclamation module 510 b may bepresent on the secondary storage system 110 b to reclaim unused space inspace-efficient secondary volumes 300 b. The space reclamation modules510 a, 510 b may operate independently from one another since resources(e.g., physical storage drives 204 a, 204 b, free storage pools, etc.)may differ on the primary and secondary storage systems 110 a, 110 b.For example, operations to reclaim unused storage space may run atdifferent times and frequencies on the primary and secondary storagesystems 110 a, 110 b since the amount of space in their free storagepools (or repository 400) may differ. A storage virtualization layer 302a, 302 b within each of the primary and secondary storage systems 110 a,110 b may expose logical primary and secondary space-efficient volumes300 a, 300 b and handle the mapping between the logical space-efficientvolumes 300 and the physical drives 204.

Referring to FIG. 7, as previously mentioned, in a synchronous datareplication system 600 such as that illustrated in FIG. 7, a writeoperation does not complete and return to the caller (e.g., host system106) until the write operation has been performed on both the primaryand secondary storage systems 110 a, 110 b. This also means thatfree-space data structures (e.g., free space descriptors 504 and/or freespace bitmaps 506) in the VTOCs 304 of the primary and secondary volumes300 a, 300 b will mirror each other as long as the primary and secondaryvolumes 300 a, 300 b are in a synchronous state. However, when freeingunused space in a space-efficient secondary volume 300 b, I/Os to theVTOC 304 of the space-efficient secondary volume 300 b may need to beheld until the process of releasing unused space can complete. Thisprevents new storage elements (e.g., tracks) from being allocated to thespace-efficient secondary volume 300 b during this period.

In order to minimize time that a VTOC 304 is locked on the secondarystorage system 110 b, logic may be added to the space reclamation module510 b. One embodiment of a space reclamation module 510 b for use on asecondary storage system 110 b is illustrated in FIG. 7. As shown, thespace reclamation module 510 b includes various sub-modules to providedifferent features and functions. These sub-modules may include one ormore of a scheduling module 700, threshold module 702, serializationmodule 704, copy module 706, unlock module 708, analysis module 710,tracking module 712, comparison module 714, supplementation module 716,release module 718, and update module 720. The modules may beimplemented in hardware, software, firmware, or a combination thereof.

The scheduling module 700 may configured to schedule a space reclamationprocess for execution on the secondary storage system 110 b. The spacereclamation process may be scheduled to occur at regular intervals(e.g., every x hours or days), at specific times of the day or week, orduring periods of reduced I/O. Additionally, or alternatively, athreshold module 702 may be configured to trigger the space reclamationprocess in response to an amount of free space in a free storage pool ofthe storage system 110 a dropping below a specific threshold.

In order to execute the space reclamation process on a space-efficientsecondary volume 300 b, the serialization module 704 may lock the VTOC304 associated with the space-efficient secondary volume 300 b so thatit can no longer be updated, and the copy module 706 may copy the VTOC304 and more particularly the free-space data structures of the VTOC 304into memory of the secondary storage system 110 b. While this lock is inplace, all I/Os to the VTOC 304 may be held or otherwise wait forexecution. Once the VTOC 304 is copied into memory, the unlock module708 may unlock the VTOC 304 to release the I/Os, thereby allowing readsand writes to once again occur to the VTOC 304.

Once the copy of the VTOC 304 is created in memory, the analysis module710 may analyze the copy rather than the VTOC 304 itself. Specifically,the analysis module 710 may analyze the copy of the free-space datastructures within the VTOC 304 to determine what storage elements (e.g.,tracks) may be released from the space-efficient secondary volume 300 b,and returned to the free storage pool. In certain embodiments, theanalysis module 710 may search for twenty-one cylinder units (a cylinderincludes fifteen tracks) that align with twenty-one cylinder divisionson the storage media, wherein a twenty-one cylinder division is agranular segment that can be released by the space reclamation module510 b. In certain embodiments, the analysis module 710 may keep track ofunused storage elements in a list 722 while performing its analysis.

While the copy is being analyzed, the tracking module 712 may monitorchanges to the VTOC 304, and more particularly to the free-space datastructures of the VTOC 304. Such changes may occur, for example, if newspace is allocated to the space-efficient secondary volume 300 b duringthe analysis. Once analysis by the analysis module 710 is complete, theserialization module 704 may lock the VTOC 304 and the copy module 706may read a new copy of the VTOC 304 into memory (or a current copy ofthe VTOC 304 if it is already in memory, or cache). The comparisonmodule 714 may then compare the new copy, and more particularly thefree-space data structures in the new copy, with the list 722 of unusedstorage elements. This comparison step may determine if the allocationof storage space or usage of storage space within the space-efficientsecondary volume 300 b changed during the analysis by the analysismodule 710. For example, during the analysis, certain storage elementsthat were formerly used may have become unused, or other storageelements that were formerly unused may have been used, or new storageelements may have been allocated to the space-efficient secondary volume300 b during the analysis. The supplementation module 716 may supplementthe list 722 of unused storage elements to reflect these changes, if anyare present.

Once the list 722 of unused storage elements is complete, the releasemodule 718 may release unused storage elements identified in the list722 from the space-efficient secondary volume 300 b and return thesestorage elements to the free storage pool. The update module 720 maythen update the VTOC 304 associated with the space-efficient secondaryvolume 300 b to reflect any released storage elements. The unlock module708 may then unlock the VTOC 304 to enable reads and writes to resumethereto. Because the majority of the analysis of the space reclamationmodule 510 b is performed while a VTOC 304 is unlocked, the spacereclamation module 510 b may minimize or reduce the amount of time thatthe VTOC 304 is unavailable for reads and writes.

Referring to FIG. 8, one embodiment of a method 800 for releasing unusedstorage space from a space-efficient secondary volume 300 b isillustrated. As shown, the method 800 initially determines 802 whetherfree space on the secondary storage system 110 b has fallen below athreshold. If so, the method 800 places 804 a first lock on the VTOC 304associated with the space-efficient secondary volume 300 b and reads 806a copy of the VTOC 304 into memory. The method 800 then releases 808 thefirst lock.

The method 800 then analyzes 810 the copy of the VTOC 304 to determinewhich storage elements in the space-efficient secondary volume 300 b arenot being used. This step 810 may include analyzing free spacedescriptor DSCBs 504 and/or the free space bitmap 506 to determine whichstorage elements (e.g., tracks) are unused. The method 800 stores 812the results of the analysis in the list 722 of unused storage elements.

The method 800 then places 814 a second lock on the VTOC 304 and reads814 a new copy of the VTOC 304 into memory. The method 800 then compares816 the list 722 of unused storage elements to the new copy to determineif any storage elements in the space-efficient secondary volume 300 bhad a change in usage status (e.g., transitioned from used to unused,unused to used, etc.), or if any new storage elements were allocated tothe space-efficient secondary volume 300 b during the analysis. If, atstep 818, the method finds differences between what is recorded in thelist 722 and what is recorded in the new copy of the VTOC 304, themethod 800 updates 820 the list 722 of unused storage elements toreflect the changes in the VTOC 304.

The method 800 then releases 822, from the space-efficient secondaryvolume 300 b, storage elements that are identified in the list 722. TheVTOC 304 associated with the space-efficient secondary volume 300 b isthen updated 824 to reflect the released storage elements. The secondlock may then be released 826. This allows any held or waiting I/Os tocomplete 828 to the VTOC 304. This, in turn, will allow acknowledgementsto be returned 830 to the primary storage system 110 a to indicate thatthe I/Os to the VTOC 304 completed successfully, which in turn willallow I/O completion acknowledgements to be returned 832 to therequester, such as the host system 106.

Referring to FIG. 9, a space reclamation module 510 in accordance withthe invention may also be used to reclaim space in space-efficientvolumes 300 in asynchronous data replication systems. One example of anasynchronous data replication system 900, comprising a primary storagesystem 110 a and secondary storage system 110 b, is shown in FIG. 9. Insuch a system 900, writes originating from a host system 106 mayinitially be written to the primary storage system 110 a, and thenmirrored to the secondary storage system 110 b as time and resourcesallow. A write is considered complete when it has completed to theprimary storage system 110 a. That is, a write acknowledgement may bereturned to the host system 106 when the write has completed on theprimary storage system 110 a but before the write has been mirrored tothe secondary storage system 110 b. The write may then be asynchronouslymirrored to the secondary storage system 110 b as time and resourcesallow.

In certain embodiments, the asynchronous data replication system 900 maybe characterized by an “idle phase,” “drain phase,” and“point-in-time-copy phase.” During the idle phase, data that is writtento the primary storage system 110 a may be asynchronously mirrored tothe secondary storage system 110 b. During this phase, updates to theprimary space-efficient volumes 300 a are recorded in a change recordingbitmap 902.

When a consistency group is created on the primary space-efficientvolumes 300 a, the change recording bitmap 902 may be converted to anout-of sync bitmap 904, and the former out-of sync bitmap 904 may beconverted to the change recording bitmap 902. This procedure that may bereferred to as a “flip.” The asynchronous data replication system 900may then enter the drain phase. During the drain phase, data in storageelements (e.g., tracks) that have been modified (as recorded in theout-of sync bitmap 904) may be copied from the space-efficient primaryvolume 300 a to the space-efficient secondary volume 300 b in order toreplicate the consistency group to the space-efficient secondary volume300 b.

When all data recorded in the out-of sync bitmap 904 has beensuccessfully copied to the space-efficient secondary volume 300 b, theconsistency group is complete and the asynchronous data replicationsystem 900 may enter the point-in-time-copy phase. During this phase, asnapshot is taken of data in the secondary space-efficient volume 300 b.This snapshot is stored as a point-in-time copy in one or more tertiaryvolumes 300 c. The tertiary volumes 300 c may reside on the same storagesystem 110 b as the secondary space-efficient volumes 300 b or on adifferent storage system 110.

As shown in FIG. 9, a space reclamation module 510 c may operate on theprimary storage system 110 a to reclaim space as needed in the primaryspace-efficient volumes 300 a. Similarly, a space reclamation module 510d may operate on the secondary storage system 110 b to reclaim space inthe secondary space-efficient volumes 300 b. These space reclamationmodules 510 c, 510 d may operate independently since resources maydiffer on the primary and secondary storage systems 110 a, 110 b.Furthermore, the space reclamation modules 510 c, 510 d may operatedifferently based on their position within the asynchronous datareplication system 900. For example, the space reclamation module 510 cat the primary storage system 110 a may operate differently than thespace reclamation module 510 d at the secondary storage system 110 b dueto the different functions occurring at each storage system 110.

FIG. 10 shows one embodiment of a space reclamation module 510 dconfigured to operate at a secondary storage system 110 b of anasynchronous data replication system 900. As shown, the spacereclamation module 510 b includes various sub-modules to providedifferent features and functions. These sub-modules may include one ormore of a scheduling module 900, threshold module 902, copy module 904,analysis module 906, determination module 908, comparison module 910,update module 912, release module 914, termination module 916, andupdate module 917. The modules may be implemented in hardware, software,firmware, or a combination thereof.

The scheduling module 900 may configured to schedule execution of aspace reclamation process on the secondary storage system 110 b. Thespace reclamation process may be scheduled to occur at regularintervals, at specific times of the day or week, during periods ofreduced I/O, or the like. Additionally, or alternatively, a thresholdmodule 902 may be configured to trigger execution of the spacereclamation process in response to an amount of free space in thespace-efficient secondary volumes 300 b, or a free storage pool, fallingbelow a specific level.

In order to execute the space reclamation process on space-efficientsecondary volumes 300 b, the copy module 904 may read a first copy ofthe VTOC 304 at the secondary storage system 110 b, and moreparticularly the free-space data structures of the VTOC 304, into memoryof the secondary storage system 110 b. The analysis module 906 may thenanalyze the free-space data structures to determine which storageelements (e.g., tracks) may be released from the space-efficientsecondary volume 300 b and returned to a free storage pool. In certainembodiments, the analysis module 906 may search for twenty-one cylinderunits that align with twenty-one cylinder divisions on the storagemedia. The twenty-one cylinder division may be a granular segment thatcan be released by the space reclamation module 510 d. In certainembodiments, the analysis module 906 may keep track of unused storageelements in a list 918 while performing the analysis.

The determination module 908 may determine when the idle phase hascompleted, which is an indicator that a consistency group has beencreated on the primary storage system 110 a. As previously mentioned, atthe end of the idle phase, the change recording bitmap 902 and out-ofsync bitmap 904 may be flipped, and data associated with the consistencygroup may be drained (i.e., copied) from the primary space-efficientvolume 300 a to the secondary space-efficient volume 300 b. At the endof the idle phase, the determination module 908 may determine, byexamining the change recording bitmap 902 (which is flipped to becomethe out-of sync bitmap 904), whether the storage element (e.g., track)storing the VTOC 304 was updated. If the VTOC 304 was not updated(indicating that the free-space data structures contained therein alsowere not updated), then the VTOC 304 at the secondary storage system 110b can be trusted to accurately describe the data on the space-efficientsecondary volume 300 b. In such a scenario, the release module 914 maybegin releasing storage elements in the list 918 of unused storageelements, since this list 918 is accurate and will not change. That is,the release module 914 may release storage elements in the list 918during the drain phase since the VTOC 304 at the secondary storagesystem 110 b will not change during the drain phase.

If, on the other hand, the change recording bitmap 902 indicates thatthe VTOC 304 was updated on the space-efficient primary volume 300 a,then the VTOC 304 on the space-efficient secondary volume 300 b may notaccurately reflect data that is stored on the space-efficient secondaryvolume 300 b. In such a scenario, the space reclamation module 510 maywait for the drain phase to complete. When the drain phase completes(indicating that the consistency group on the space-efficient primaryvolume 300 a has been successfully replicated to the space-efficientsecondary volume 300 b), the VTOC 304 on the space-efficient secondaryvolume 300 b will accurately describe data on the space-efficientsecondary volume 300 b. At this point, the copy module 904 may read asecond copy of the VTOC 304 on the space-efficient secondary volume 300b into memory. The comparison module 910 may then compare the secondcopy of the VTOC 304, and more particularly the free-space datastructures in the second copy, to the list 918 of unused storageelements.

The comparison step may determine if the allocation of storage space orusage of storage space within the space-efficient secondary volume 300 bchanged during the analysis by the analysis module 906. For example,during the analysis, certain storage elements that were formerly usedmay have become unused, or other storage elements that were formerlyunused may have become used, or new storage elements may have beenallocated to the space-efficient secondary volume 300 b. The updatemodule 912 may update the list 918 of unused storage elements to reflectany such changes. In certain embodiments, the supplementation module 912may update the list 918 by removing, from the list 918, any unusedstorage elements that are associated with changes to the VTOC 304. Thisis due to the fact that some storage elements that were formerly unusedmay have become used during analysis by the analysis module 906.

Once the list 918 of unused storage elements is complete, the releasemodule 914 may release unused storage elements identified in the list918 from the space-efficient secondary volume 300 b and return thesestorage elements to the free storage pool. In certain embodiments, thisrelease may be performed during the point-in-time-copy phase of theasynchronous data replication process. If the point-in-time-copy phaseends before all space has been released from the space-efficientsecondary volume 300 b, the termination module 916 may terminate therelease process even if it has not yet completed. This will ensure thatthe release process does not extend past the point-in-time-copy phaseand possibly into the next idle phase. This will also prevent delays ofthe next idle phase (thereby avoiding a performance impact) and notadversely affect the recovery point.

Referring to FIG. 11, one embodiment of a method 1100 for releasingstorage space in an asynchronously mirrored space-efficient secondaryvolume is illustrated. As shown, the method 1100 initially determines1102 whether free space on the secondary storage system 110 b has fallenbelow a threshold. If so, the method 1100 reads 1104 a first copy of theVTOC 304, and more particularly a first copy of the free-space datastructure in the VTOC 304, into memory. This may be accomplished byreading the track or other storage element that contains the VTOC 304into memory. The method 1100 then analyzes 1106 the first copy todetermine which storage elements in the space-efficient secondary volume300 b are not being used. This step 1106 may include analyzing freespace descriptor DSCBs 504 and/or a free space bitmap 506 in the VTOC304 to determine which storage elements (e.g., tracks) are unused. Themethod 1100 stores 1108 the results of this analysis in the list 918 ofunused storage elements.

The method 1100 then waits 1110 for the idle phase to complete(indicating that a consistency group is complete on the space-efficientprimary volume 300 a), if it has not already completed. When the idlephase completes, the method 1100 determines 1112 whether the free-spacedata structure in the VTOC 304 was changed at the primary storage system110 a. This may be accomplished by analyzing the change recording bitmap902 (which converts to the out-of sync bitmap 904 upon initiating thedrain phase). Specifically, this may be accomplished by analyzing thebit in the change recording bitmap 902 that documents updates to thestorage element (e.g., track) storing the VTOC 304. If the free-spacedata structure has not changed, then the method 1100 may immediatelybegin 1114 to release, from the space-efficient secondary volume 300 b,storage elements identified in the list 918 of unused storage elements.This is because the VTOC 304 on the secondary storage system 110 b willnot change during the drain phase and the VTOC 304 on the secondarystorage system 110 b accurately represents data on the space-efficientsecondary volume 300 b.

If, at step 1112, the free-space data structure has changed at theprimary storage system 110 a, the method 1100 waits 1116 for the drainphase to complete. This allows the VTOC 304 to be copied over to thesecondary storage system 110 b. When the drain phase completes, themethod 1100 reads 1118 a second copy of the VTOC 304 on thespace-efficient secondary volume 300 b. The method 1100 then compares1118 the list 918 of unused storage elements to the second copy of theVTOC 304 to determine if any storage elements in the space-efficientsecondary volume 300 b had a change in usage status (e.g., transitionedfrom used to unused, unused to used, etc.), or if any new storageelements were allocated to the space-efficient secondary volume 300 bduring the analysis 1106. If, at step 1120, the method finds 1120differences between what is recorded in the list 918 and what isrecorded in the second copy of the VTOC 304, the method 1100 updates1122 the list 918 of unused storage elements to reflect the changes. Incertain embodiments, updating 1122 the list 918 may include removing,from the list 918, any unused storage elements that are associated withchanges to the VTOC 304 (since these unused storage elements may havebecome used).

The method 1100 then determines 1124 whether the point-in-time-copyphase has completed. If the point-in-time-copy phase has not completed,the method 1100 either begins 1126 to release storage elementsidentified in the list 918 (if this process has not already started) orcontinues 1126 to release storage elements identified in the list 918if, for example, the release process already started at step 1114. Thisrelease process continues until all storage elements identified in thelist 918 are released from the space-efficient secondary volume 300 b,or the point-in-time-copy phase completes. If the point-in-time-copyphase completes before all space has been released, the method 1100 ends1128 the release process. If the release process finishes before thepoint-in-time-copy phase completes, the method 1100 simply ends 1128.

The flowcharts and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowcharts or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the Figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. Other implementationsmay not require all of the disclosed steps to achieve the desiredfunctionality. It will also be noted that each block of the blockdiagrams and/or flowchart illustrations, and combinations of blocks inthe block diagrams and/or flowchart illustrations, may be implemented byspecial purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

1. A method for releasing storage space in asynchronously mirroredspace-efficient secondary volumes, the method comprising: reading afirst copy of a free-space data structure stored on a space-efficientsecondary volume, the free-space data structure tracking usage status ofstorage elements in the space-efficient secondary volume; analyzing thefirst copy to determine which storage elements in the space-efficientsecondary volume are not being used; waiting for a consistency group onthe space-efficient secondary volume to complete; upon completion of theconsistency group, reading a second copy of the free-space datastructure; comparing the first copy to the second copy to determinewhich storage elements had their usage status change during analysis ofthe first copy; and releasing storage elements in the space-efficientsecondary volume that are not being used.
 2. The method of claim 1,wherein releasing storage elements comprises releasing storage elementsduring creation of a point-in-time copy of the space-efficient secondaryvolume.
 3. The method of claim 2, further comprising ceasing to releasestorage elements upon completion of the point-in-time copy.
 4. Themethod of claim 1, wherein the storage elements are tracks.
 5. Themethod of claim 1, wherein the free-space data structure is a bitmapcomprising a bit for each track in the space-efficient secondary volume.6. The method of claim 1, wherein the free-space data structure iscontained within a volume table of contents (VTOC) stored on thespace-efficient secondary volume.
 7. The method of claim 1, whereinanalyzing the first copy comprises generating a list of unused storageelements in the space-efficient secondary volume.
 8. A computer programproduct to release storage space in asynchronously mirroredspace-efficient secondary volumes, the computer program productcomprising a non-transitory computer-readable storage medium havingcomputer-usable program code embodied therein, the computer-usableprogram code comprising: computer-usable program code to read a firstcopy of a free-space data structure stored on a space-efficientsecondary volume, the free-space data structure tracking usage status ofstorage elements in the space-efficient secondary volume;computer-usable program code to analyze the first copy to determinewhich storage elements in the space-efficient secondary volume are notbeing used; computer-usable program code to wait for a consistency groupon the space-efficient secondary volume to complete; computer-usableprogram code to, upon completion of the consistency group, read a secondcopy of the free-space data structure; computer-usable program code tocompare the first copy to the second copy to determine which storageelements had their usage status change during analysis of the firstcopy; and computer-usable program code to release storage elements inthe space-efficient secondary volume that are not being used.
 9. Thecomputer program product of claim 8, wherein releasing storage elementscomprises releasing storage elements during creation of a point-in-timecopy of the space-efficient secondary volume.
 10. The computer programproduct of claim 9, further comprising computer-usable program code tocease to release storage elements upon completion of the point-in-timecopy.
 11. The computer program product of claim 8, wherein the storageelements are tracks.
 12. The computer program product of claim 8,wherein the free-space data structure is a bitmap comprising a bit foreach track in the space-efficient secondary volume.
 13. The computerprogram product of claim 8, wherein the free-space data structure iscontained within a volume table of contents (VTOC) stored on thespace-efficient secondary volume.
 14. The computer program product ofclaim 8, wherein analyzing the first copy comprises generating a list ofunused storage elements in the space-efficient secondary volume.
 15. Asystem to release storage space in asynchronously mirroredspace-efficient secondary volumes, the system comprising: at least oneprocessor; at least one memory device operably coupled to the at leastone processor and storing instructions for execution on the at least oneprocessor, the instructions causing the at least one processor to: reada first copy of a free-space data structure stored on a space-efficientsecondary volume, the free-space data structure tracking usage status ofstorage elements in the space-efficient secondary volume; analyze thefirst copy to determine which storage elements in the space-efficientsecondary volume are not being used; wait for a consistency group on thespace-efficient secondary volume to complete; upon completion of theconsistency group, read a second copy of the free-space data structure;compare the first copy to the second copy to determine which storageelements had their usage status change during analysis of the firstcopy; and release storage elements in the space-efficient secondaryvolume that are not being used.
 16. The system of claim 15, whereinreleasing storage elements comprises releasing storage elements duringcreation of a point-in-time copy of the space-efficient secondaryvolume.
 17. The system of claim 16, wherein the instructions furthercause the at least one processor to cease to release storage elementsupon completion of the point-in-time copy.
 18. The system of claim 15,wherein the storage elements are tracks.
 19. The system of claim 15,wherein the free-space data structure is a bitmap comprising a bit foreach track in the space-efficient secondary volume.
 20. The system ofclaim 15, wherein the free-space data structure is contained within avolume table of contents (VTOC) stored on the space-efficient secondaryvolume.